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A Review of Voucher Program Studies, 1998 
Cleveland Public Schools 



Introduction 

Since the initiation of the Cleveland Scholarship and Tutoring 
Grant Program (CSTGP) during the summer of 1996, much nationwide 
attention has been focused upon the program and its progress. To date, 
the following chronology highlights the past thirty months of the choice 
school program in the Cleveland, Ohio. 

1995: The "Ohio Pilot Scholarship Program" was included in the Ohio 
State budget specifying that nearly 2,000 primary grade children in the 
Cleveland City School District could be chosen to receive scholarships 
to attend private and/or religious schools within the district. 

June, 1996: Franklin County Common Pleas Court declared the "pilot 
voucher program" constitutional following a legal challenge by program 
opponents including the American Federation of Teachers, the National 
Education Association, and other agencies. 

Fall, 1996: About 2,000 students in grades kindergarten through three 
were selected to receive scholarships and attended "choice" schools. 

May, 1997: The Tenth District Court of Appeals overturned the previous 
ruling and declared that the program was unconstitutional. However, 
the State of Ohio sought to continue the program pending a further 
appeal to the Ohio Supreme Court. The first year of choice school 
participation was completed by nearly 2,000 students. 

June, 1997: A news release by Ohio governor George Voinovich and David 

Brennan (founder of HOPE Academy, one of the "voucher" schools) cites 
the fall-to-spring performance gains of students in these schools. 

July, 1997: The American Federation of Teachers (AFT) released its study 
of the "Cleveland Voucher Program: Who Chooses, Who's Chosen, Who Pays?" 
by Dan Murphy, F. Howard Nelson, and Bella Rosenburg. This report 
examined the first year implementation of the plan, addressed who 
applied for vouchers, who used them, what schools they attended, and 
what was the cost of the program. 

August, 1997: A second cohort of voucher participants began the new 

academic year. Participation levels increased slightly under the 
formulated program expansion for the 1997-98 school year. 

Approximately 1000 additional students were enrolled under the program. 

September, 1997: The Program on Education Policy and Governance (PEPG) 

released its report "An Evaluation of the Cleveland Scholarship 
Program" by Jay Greene, William Howell, and Paul Peterson. This report 
included the HOPE school test score analysis (cited in the June news 
release) and the results of a parent survey conducted by the University 
of Northern Illinois. Voucher applicants were surveyed concerning their 
attitudes towards school and the voucher program. 
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November, 1997: The Public Policy Forum (PPF) released its report: 

"School Choice in Cleveland and Milwaukee: What parents Look For" by 
Emily Van Dunk- This survey analysis represented the initial phase of 
an effort to focus on the views of parents, teachers, and administrators 
on one aspect of accountability: what parents look for in a school. 

November, 1997: The Junior Achievement Evaluation Project (later called 

the Indiana Center for Evaluation, Indiana University) forwarded its 
first year (1996-97) project evaluation report. The report was the 
formal program evaluation of the Cleveland Scholarship (voucher) Program 
contracted under the law by the Ohio State Department of Education. "A 
Comparative Evaluation of the Cleveland Scholarship and Tutoring Grant 
Program" by Kim Metcalf, William Boone, Frances Stage, Todd Chilton, and 
Patty Muller specifically addressed two questions: what are the program 
effects on student achievement and, what are the characteristics of 
participants and their families and how do they compare to those of the 
non-participants? The report was publicly released in March, 1998 after 
a delay by the contracting agency (the Ohio Department of Education). 

February, 1998: The Public Policy Forum released the second phase of 

its report, "Choice School Accountability: A Consensus of Views in Ohio 
and Wisconsin" by Emily Van Dunk. The study cited and recommended what 
information, prepared by whom, and what compliance methods, should guide 
private school reporting for accountability to the public. 

April - August, 1998: Public debate (sensationalized in the media) 

ensued involving the HOPE school supporters /researchers , the American 
Federation of Teachers (Cleveland Teachers Union), and the IU evaluators 
concerning aspects of their respective studies. 

Summer, 1998: Awards of the third round of vouchers were completed for 

the the 1998-99 school year. About 1300 new voucher recipients were 
enrolled in private schools. The current school year finds a total of 
3,744 voucher recipients (from all three years) enrolled in the program 
utilizing voucher funding. 3,030 students are enrolled in religious 
schools, 713 are attending non-sectarian schools. 

November, 1998: The Indiana Center for Evaluation released its "Second- 
Year Report: Evaluation of the Cleveland Scholarship Program" for the 
1997-98 school year. This research effort sought to answer the 
following questions: 

1. Are there differences between students who returned to the 
scholarship program during their fourth grade year and 
those who did not return after their third grade year? 

2. Are there differences between fourth grade scholarship and 
public school students with regard to demographic and 
background characteristics of preprogram achievement? 

3. Are there differences in classroom relevant variables 
(i.e., class size, teachers' education level, and teachers' 
experience) between scholarship classes and public school 
classes? 
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4. What are the effects of the scholarship program on 
students ' academic achievement after two years in the program 
and when other relevant variables are controlled? 

Each of these reports and events were completed under the auspices 
of different agencies; the highlighted findings that follow may reflect 
politically inspired perspectives contained in the conclusions which are 
cited. 

The Studies: Findings, Conclusions, Observations, Questions, and Issues 

In general the studies mentioned above investigated three aspects 
of the voucher program's first three years of operation: program 
implementation methods and participants, parental attitudes towards the 
opportunity and program, and achievement progress of the participants 
and their non-participant counterparts. The following discussion 
includes highlights of the studies with comments and issues pertaining 
to the studies . 

The Program on Educational Policy and Governance Report 

The PEPG Report summarized survey responses from parent interviews 
conducted by PEPG staff and by the staff of Northern Illinois 
University. In all, nearly 200 interviews were conducted among voucher 
recipients and those who did not enroll in the program. Also, student 
achievement results from one voucher school (HOPE Academy) was reported. 
Among the highlighted findings were: 

1. Parents of scholarship recipients who previously attended 
public schools were much more satisfied with every aspect of 
their choice school than applicants who did not receive a 
scholarship, but attended public schools instead. 

2. Choice schools did well at retaining students in the 
program. 

3. Non-participant parents cited transportation, financial 
factors, and admission to a preferred public school most 
frequently as reasons for not accepting a voucher. 

4. Reasons for applying for a voucher school assignment 
were: academic quality, greater safety, location, religion, 
and friends among public school recipients. 

5. The average family income of voucher recipients from 
public schools was less than that of non-recipients who 
remained in public schools. 

6. The survey data was limited; due to the respondent 
identification process, voucher recipients could be reached 
more easily and share information more readily than non- 
recipients . 

7. Test score results in mathematics and reading show large 
gains for CSTGP students attending the two schools (HOPE) 
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established in response to the creation of the CSTGP. 

8. In sum, both parental survey and initial test score 
results provide strong justification for the legislative 
decision to continue the program. 

Questions and Issues: The PEPG Report. 

The survey results are suspect by the authors' own admission (see 
statement #6 above) . Participants would be likely to respond positively 
about their experience having been selected to receive a voucher. 

The report indicates that the students in the HOPE schools were 
administered the CAT test in the fall and spring of the 1996-97 school 
year. The version, form, and level of the test are not specified, as 
well as how the test was scored (using fall and spring norms?) to assess 
the gain. 

Also, it is interesting that the results used for reporting are 
indicated as average Percentile Rank scores which are subsequently 
added/subtracted to report a gain score for HOPE students. Percentile 
rank scores are rank order scores and are not suitable for arithmetic 
operations. If, indeed, the "average percentile ranks" were accurately 
derived from scaled score averages, they may only be assessed for gains 
by utilizing an NCE (normal curve equivalent) score. For example, 
converting the reported results on Table 4.2 (page 87 of the report) to 
NCE scores and analyzing the gains in reading, language, mathematics, 
and mathematical concepts would yield gains of 3.4, -2.5, 6.6, and 7.3 
points respectively (versus those posted "PR" gains). To put this in 
context, the last time Chapter I/Title I students in the Cleveland 
Schools were assessed using a fall to spring pre-post model, targeted 
gains were expected to be 7 NCE's or greater. Seven NCE units was the 
normal anticipated outcome for progress in reading and mathematics for 
students provided with compensatory services (which are much like those 
students that the author describes as the HOPE attendees ) . None of the 
gains reported reach the seven NCE level, which indicates that normal 
progress throughout the course of an academic year was evident among 
HOPE students , but not large gains . 

In the text of the report (pages 40 - 42) the authors refer to one 
to two percentile point decline in reading scores among district 
students. Interestingly, the district does not report percentile scores 
in its bulletins or profiles. All scores are reported in NCE units. 

The authors are likely referring to the following information 
available from Test Bulletin #1 concerning the spring, 1996 reading 
achievement results (from the California Achievement Test, Form E). 
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Cleveland Public Schools 
Spring Reading Achievement 



GRADE 


1995 

NCE 


1996 

NCE 


Corresponding Percentiles 
( not reported in TB#1 ) 


1 


49.6 


47.8 


49.8 


48.8 


2 


46.1 


44.8 


49.9 


49.4 


3 


48.8 


47.5 


49.4 


48.7 



A quick review of the chart indicates declines in performance among 
students from 1995 to 1996; however, reading the chart diagonally (grade 
one to two to three) indicates a decline in reading scores between first 
and second grade but an increase in performance from second to third 
grade. Note the corresponding percentile rank scores at this level tell 
a different story; this illustrates the reason why percentile rank 
scores cannot be used to analyze performance gains. Notably, despite 
the inconsistencies between the two metrics, gains or declines were all 
less than one point, not between one and two percentile points as noted 
by the PEPG authors . 

Finally, information exchanged with the Junior Achievement 
Evaluation Project (Center for Evaluation, Indiana University) has been 
combined with achievement test results available earlier (1996) for 
students who subsequently enrolled in voucher schools. The file 
contains 749 records of students who were in the second grade in the 
district in 1996. The reading scores of those students indicated a 46.3 
NCE average for the reading comprehension subtest among all voucher 
participants. The 46 students within this group who enrolled in the 
HOPE schools posted a 45.2 NCE average. During that year, all Cleveland 
second graders (including these 749 students) averaged 44.8 NCE's on the 
same subtest. While HOPE schools received students with slightly lower 
reading achievement scores than other voucher participants, the entire 
population of 1997 third grade voucher participants averaged higher 
scores than their district counterparts as second graders. 

Additional debate about differing research results ensued during 
1998. See the discussion below concerning other technical issues. 

The American Federation of Teachers Report 

The AFT Report investigated the implementation of the first year 
of voucher awards. The authors concluded: 

1 . The voucher program did not appreciably increase the 
educational choices available to parents of students of 
"failing" public schools including an analysis of those who 
participated, those who did not, and the participation in 
tutoring grants within the school district. 

2. About one-half of the public school students who were 
eligible for a voucher did not enroll in private schools. 

3. Voucher students who were previously enrolled in private 
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schools received preferred placement; nearly half of the 
voucher students from public schools were enrolled in newly 
formed schools . 

4. The public cost of the voucher program is greater than 
believed; moreover, the program was mostly paid for out of 
state aid designated for the Cleveland Public Schools. 

5. Taxpayers spent $1.6 million to fund vouchers for 496 
students who were already enrolled in private schools and 
remained in them. 

6. Taxpayers spent another $1.7 million to fund vouchers for 
525 former public school students to enroll in four new 
schools with no educational or financial track record. 

7. In the more established private schools, the total public 
cost per voucher far exceeded the amount of money actually 
being spent on new voucher students. 

Issues and questions for the AFT Report 

The specific data utilized by the AFT was provided by the CSTP 
office (enrollment information), the Cleveland City School District 
(financial information), and was derived from the text of the law which 
created the program. The ensuing analysis addressed issues inherent in 
the intent of the law: to fund a small number (about 2,000) of low 
income students at private schools and to enable an equal number of 
Cleveland Public Schools students to receive tutoring grants. The 
conclusions noted above addressed the designed inquiry. 

The results are noteworthy, especially when taken in context with 
the other studies reported here. A realization of the actual 
participation patterns and characteristics of the participants provides 
an interesting perspective for review of the other studies noted. 

The Public Policy Forum Report 

The Forum Report, Phase One, sought to determine what mattered to 
parents when choosing a school for their child. About 270 parents, 
teachers, and administrators were interviewed to provide information for 
these conclusions: 

1. Information about a school's program (i.e. curriculum and 
instruction) is the most common piece of information parents 
want when deciding upon a school for their child. 

2. Information about teachers is next. 

3. Other criteria in order of preference: school 
characteristics (class size and student body composition), 
general student outcomes (life skills and promotions rates), 
safety and discipline, standardized test scores, level of 
parent involvement, and school reputation. 




7 



8 



4. Teachers and administrators concur with parents (i.e. 
finding #1 ) . 

5. There is essential agreement among parents, across public 
and private schools , on the criteria that are important to 
them in selecting a school. 

The Forum Report, Phase Two extended the discussion of what 
parents want to know about their children' schools. Having identified 
what information is most desirable, this phase of the research sought 
to identify specifics of what information to report, who should 
compile and report this information, and how reporting should be 
monitored and enforced. The following conclusions were reported in 
phase two. 



1. Choice schools should publicly report mission; philosophy; 
governing structure; curriculum and teacher methods; the 
qualifications of teachers and administrators; pupil 
achievement on standardized tests; how money is budgeted and 
spent; and the attendance, graduation, and suspension rates. 

2. A public board with representatives of private choice 
schools and public schools should be created to gather and 
report the required information for the public. 

3. Choice schools will comply with a one year probationary 
period; further non-compliance should result in loss of 
public funds. 

Issues and Questions for the Forum Reports 

These reports are unique to the others in two ways . The 
information gathered for the study was derived from both Cleveland, Ohio 
and Milwaukee, Wisconsin where school choice programs are in effect. 
Also, the methodology employed by the researchers utilized open-ended, 
non-structured interview strategies, thereby eliciting responses which 
were not prompted. The results, therefore, become interesting from the 
point of view that no artificial dichotomies were imposed upon the 
respondents . 

Responses concerning the reasons for the school selection do 
reinforce, to some degree, findings cited in the PEPG Report above (i.e. 
school program and safety) . 

Further, the findings and recommendations of phase two add a 
practical dimension to the issues. Should these programs, and others, 
remain in operation, accountability measures should require schools to 
conform to those measures imposed upon existing private and public 
schools . 

The Indiana University Reports 

The IU Report (Year One) of voucher participant versus non- 
participants provides the most comprehensive examination of student 
achievement patterns completed to date. Among the observations: 
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1. Scholarship students who previously attended a Cleveland 
public school were achieving at slightly higher levels before 
they entered the program than students who remained in CPS, 
however there are no significant differences between these 
groups of students in eligibility for free lunch, gender, 
race, or parental living arrangements. 

2. After eight months in the program, there appear to be no 
statistically significant differences in the adjusted third 
grade achievement between scholarship and non-scholarship 
public school students when background and demographic 
factors are included in the analysis . 

3. The results suggest that the scholarship program does not 
draw Cleveland's "best" students; scholarship participants 
look very much like their non-participant peers in many ways. 

4. Early results indicate that the first year of the 
scholarship program led neither to significantly greater nor 
lower achievement for students who participated. 

The report concludes with a discussion concerning the limitations 
of the study at hand and of the issues and problems present which might 
impact the findings, as well as recommendations for future evaluation 
plans. All in all, this study is the most comprehensive of those cited 
so far; methodologically speaking, it provides the best analysis of the 
student achievement among participant and non-participant control group 
students . 

Issues and Questions for the IU Report 

The analysis included a report of third grade achievement data 
contributed by the HOPE schools along with district-contributed grade 
two data for a limited number of students . The data was not integrated 
with other program and non-participant data since the test forms and 
testing conditions varied for the HOPE students. Conclusions were not 
drawn concerning the HOPE voucher participants; results were reported in 
parallel where they could be discussed. 

Debated Issues: Student achievement among participant and non- 
participant students. 

Shortly after the formal release of the IU evaluation report, 
reactions and challenges filled the media; in particular, Paul Peterson 
and Jay Greene (PEPG) questioned the methodology and analyses 
techniques utilized by the Indiana team. Specifically, they questioned 
the interim analytical techniques, the "legitimacy" of the comparison 
(CPS non-participants) group, and the reliability of second grade scores 
(of all study participant) used in the analysis. 

PEPG took its challenges to the editorial pages of the Wall Street 
Journal, The Washington Post, the Cleveland Plain Dealer, and Education 
Week to raise these technical issues. In response, the IU evaluation 
team conducted additional analyses and provided the following results. 
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First, in response to the criticism of analytical methodology, the 
IU team re-analyzed the data from the first year evaluation utilizing 
the multiple regression technigues suggested by the PEPG team. Results 
of this re-analysis provided identical results as the ANCOVA's performed 
initially: the scholarship program status was not statistically 
significant in predicting any of the third grade scores; the level of 
significance was identical for both ANCOVA and seguential regression. 

Secondly, an analysis to compare characteristics of the Cleveland 
Schools used in the study (n=28) versus those not included (n=34) found 
no significant differences between schools included versus those not 
included. Again, MANOVA determined these results when characteristics 
of prior test scores (NCE) , enrollment levels, race and gender 
composition (students and staff), student mobility, attendance, poverty 
rates, and teacher/student ratios were included in the analysis. PEPG's 
claim of non-comparability of control group schools was dispelled. 

Finally, PEPG claimed in all of its editorial letters that the 
Cleveland Public Schools' second grade test scores were implausible 
(that they were higher than the previous year) and were not 
independently proctored. "Clearly, PEPG says, the second grade scores 
used by IU as a benchmark were inflated". The Indiana team again 
responded with a review of the test's documentation which indicated that 
second grade test scores were not unreliable based upon the second to 
third grade achievement score correlations attained in the study. Also, 
additional information is presented below from analyses of the district 
achievement data. 

Additional Pertinent Information and Data 

During the course of providing reguested information from the 
Cleveland School District files to the various agencies, several 
summaries and comparative data analyses were completed within the 
district concerning the samples utilized by the evaluators. The 
following descriptive data is worth noting. 

1996 Grade 2 California Achievement Test Results 
1995-96 Academic Year 

Population Vocabulary Comprehension N 

Mean NCE Mean NCE 



Voucher St's 51.5 46.4 699 
HOPE Students 48.8 45.2 46 
All CPS St's 48.9 44.8 5907 



This information reinforces the IU observation that those voucher 
students who enrolled at the HOPE schools are slightly lower achieving 
students than the entire voucher population. It also provides an 
interesting piece of information not reported in either study which 
addressed the HOPE student population. 

Notwithstanding the issues cited above (with the PEPG report) , 
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this summary of the 46 HOPE students with available data from their 
second grade year in Cleveland indicates a 45.2 NCE average for reading 
comprehension as second graders on the California Achievement Test, Form 
E. The IU report cites a third grade achievement level among 36 HOPE 
students at 40.9 NCE's, a greater than four point decline since their 
second grade year. Despite the fact that these are NOT matched scores, 
the same logic concerning the "normal progress" concept underlying the 
NCE score would indicate a zero (0.0) NCE change from one year to the 
next on the same achievement battery. This fact does refute the claims 
of large achievement gains made in the PEPG report. 

Finally, to address a discussion among the authors of the PEPG and 
IU reports concerning the stability of the district's second grade CAT 
scores over time, the following analysis was compiled. 

Grade 2 California Achievement Reading Test Scores 
Cleveland Public Schools Students 
Standardization Sample 



Year 


N 


Vocabulary 
Mean NCE 


(sd) 


Comprehension 
Mean NCE (sd) 




1997 


5907 


48.9 


(24.3) 


44.8 


(20.2) 


CPS 


1996 


5593 


51.8 


(24.0) 


46.1 


(19.5) 


CPS 


1995 


5452 


55.0 


(23.6) 


48.1 


(19.7) 


CPS 


1986 


20642 


45.0 


(23.0) 


45.0 


(21.0) 


STD* 



*includes urban student participants with identical results 

A review of this chart indicates a declining achievement level 
among second graders over the past three years. However, a review of 
the standard deviation values illustrates a very consistent pattern over 
the three years of CPS data as well as with the original standardization 
data published by CTB-McGraw Hill, the test publisher. This does 
indicate a stable distribution about the mean of the tested population 
over three sampled years and points to the reliability of this 
instrument. 

IU Evaluation Report: Year Two (1997-98) extended the analysis of 
student characteristics and achievement during the second year of 
voucher program operation. Notably, this study included an analysis of 
HOPE school students who were included in the program achievement 
testing. The findings from year two: 

1. Returning and non-returning program students do not differ 
significantly on any background or demographic , factors , or 
preprogram achievement measures. However, students who 
remained at voucher schools achieved significantly higher 
scores in reading, science, and social studies than those who 
did not remain after the third grade. 

2. Program participants and their public school counterparts 
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are remarkably similar in terms of background demographic 
characteristics (gender, race, and poverty ) and previous 
achievement (unlike year one) . 

3. The classrooms of participant and non-participants 
differed significantly; class sizes (smaller in public 
schools), teacher advanced training (more in public schools), 
and teacher experience (greater in public schools). 

4. When demographic, classroom, and prior achievement factors 
are controlled, significant differences are evident with 
higher voucher student achievement in language, however, no 
significant differences are evident in reading, science, 
mathematics, and social studies. Further, however, it was 
found that HOPE students achieved significantly lower on all 
measures (subjects) of fourth grade performance than their 
public (non-participant) and other private (voucher) 
counterparts . 

The second year study, while responding to more detailed 
evaluation questions, reinforces the first year results. 

Participants and non-participants share similar background 
demographic characteristics; their settings (school environments) 
vary significantly; but their achievement does not (except in 
language and with selected grouping analyses). As the authors 
conclude, only additional study over time will determine any 
significant or long term effects of choice school placements. 

Conclusions 

Several studies of the Cleveland voucher experience focused upon 
varied aspects of the program. Parent satisfaction levels were 
assessed, school choice opinions were gathered, student selection and 
participation patterns were documented, and finally, student achievement 
was analyzed. At this point, the following statements can be made. 

1. In general, it does not appear that voucher participants 
demonstrate achievement gains (as a result of attending a private 
school) any greater than their public school counterparts. Significant 
differences were not found to exist when acknowledging background 
factors of students in the study except higher language achievement 
among program participants than public school students during year two. 

2. Satisfaction and selection reasons (from two different studies) 
appear to explain each other. When parents are provided a choice and 
accept it, they tend to be satisfied with that choice. Those who did 
not accept the option were less likely inclined to respond to the 
survey; those who had no choice in the first place would likely be less 
satisfied not having been given the option. 

3. Parents of public and private school students alike, appear to 
pay more attention to non-achievement school factors when asked what 
they look for in a school and how they should obtain information. While 
achievement information is specified, several other factors are 
considered a higher priority. 
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It appears, after two years of evaluation and debate, that 
academic achievement is not clearly impacted (positively or negatively) 
by participation in a voucher program. Several factors impact the 
experience and satisfaction of students and parents in a school setting. 
As Cecilia Rouse has reported, "not all public schools are created 
equal; additionally, not all private schools are created equal" 
following her analysis of several regular, choice, and magnet schools in 
Milwaukee. A specific program's value appears to be in the satisfaction 
of choice among participants and non-participants alike. When one is 
"comfortable" with a situation or circumstance, one is likely to 
function and perform without distraction. 
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